Providing dynamic serviceability for software-defined data centers

ABSTRACT

Examples described herein include systems and methods for providing dynamic serviceability for a software-defined data center (“SDDC”). An example method can include collecting data-center metrics from a management service that monitors the SDDC, filtering the data-center information based on a predetermined list of metrics provided by a partner entity, and translating the filtered data-center information into a partner-specific format requested by the partner entity. The example method can also include generating metadata associated with the translated data-center information and transmitting the metadata and translated data-center information to a partner site associated with the partner entity. If the partner site is not available, the method can include transmitting the information to a partner-accessible storage location and, when the partner site becomes available, identifying the storage location and failed attempt to deliver the information.

RELATED APPLICATIONS

Benefit is claimed under 35 U.S.C. 119(a)-(d) to Foreign ApplicationSerial No. 201841021360 filed in India entitled “PROVIDING DYNAMICSERVICEABILITY FOR SOFTWARE-DEFINED DATA CENTRES”, on Jun. 7, 2018, byVMware, Inc., which is herein incorporated in its entirety by referencefor all purposes.

BACKGROUND

Traditional data centers typically contain multiple servers, each ofwhich can be limited to using its own processor(s) and memory along witha shared, central storage location. Using virtualization techniques,these servers can pool their computing, memory, and storage resourceswhile each executing multiple virtual machines, increasing the outputand utilization of the data center. The system can utilize off-the-shelfcomponents with direct-attached storage, providing an administrator withthe ability to plug and play into a pool of similar components. Anadministrator can access both the hardware and software resourcesthrough a single platform. A data center utilizing a fullysoftware-defined infrastructure, including virtualized computing,storage, and networking, can be considered a “hyper-convergedinfrastructure” (“HCl”) or “software-defined data center” (“SDDC”).

While SDDCs provide many benefits, many of the current solutions fail toprovide adequate serviceability tools. An SDDC may utilize hardware andfirmware from various partners, resulting in a variety of serviceabilityneeds. Many SDDCs lack the ability to (1) provide a proactive andvendor-agnostic service, (2) associate serviceability with maintenanceby locking or “muting” a resource to avoid spurious issues, (3)automatically configure telemetry processing pipelines to suit apartner's needs, (4) automatically detect and fix problems, (5) providean interface for partners to implement patches or other fixes, and (6)provide multi-site SDDC support for serviceability.

As a result, a need exists for systems and methods for providingimproved, dynamic serviceability for data-centers, including SDDCs.

SUMMARY

Examples described herein include systems and methods for providingdynamic serviceability for an SDDC. An example method includescollecting data-center metrics from a management service that monitorsthe SDDC. The data-center metrics can include statistics, events,alarms, and alerts generated by software or hardware components of theSDDC.

The example method can also include filtering the data-centerinformation based on a predetermined list of metrics provided by apartner entity. As used herein, a “partner entity” or “partner” can bean organization, company, group, individual, or service that isresponsible for managing service requests associated with an SDDC or aportion of an SDDC.

The example method can also include translating the filtered data-centerinformation into a partner-specific format requested by the partnerentity. For example, the partner entity can request the data-centerinformation to be provided in the form of a service request. Theservice-request format can be specific to the partner entity andprovided in advance. The step of translating can include formatting thefiltered data-center information such that it matches a service-requestformat specific to the partner entity.

The example method can include generating metadata associated with thetranslated data-center information. The metadata can include any type ofinformation that describes or is relevant to the translated information.For example, it can describe the time or times at which the informationwas collected, the type of filtering or translation applied to theinformation, and the size of the data embodying the information. Theexample method can include transmitting the metadata and translateddata-center information to a partner site associated with the partnerentity.

In one example, the method can also include, in response to determiningthat the partner site is unavailable, forwarding the metadata andtranslated data-center information to a partner-accessible storagelocation. The storage location can be a cloud storage location, forexample. The method can also include determining that the partner sitehas recovered. In response, the method can include generating an alertregarding downtime of the partner site and identifying the destinationof the metadata and translated data-center information. The partnerentity can use that information to retrieve the information from thestorage location.

The collection of data-center metrics can be performed according to apoll time. The poll time can be dynamically set by an inference enginebased on settings provided by the partner entity. For example, thepartner entity can specify conditions necessitating more frequent (orless frequent) polling times, and the inference engine can recognizethose conditions and implement the relevant polling time.

The stages of the example method can be performed based on a least onerule supplied by the partner entity through a serviceabilityconfiguration interface and enforced by an inference engine. Theinference engine can be a rules engine that incorporate artificialintelligence, such as by executing machine-learning algorithms.

The example method can also include receiving a response from a partnerentity through a serviceability operations interface and implementing asolution at the SDDC, based on the received response, without manualintervention. For example, an inference engine can detect a problem,interpret instructions related to solving that problem, and carry outthe instructions without requiring an administrator's input. Theinference engine can execute a machine-learning model provided by thepartner entity to make decisions on actions to take, such as restartinga service or system that has produced an error.

The examples summarized above can each be incorporated into anon-transitory, computer-readable medium having instructions that, whenexecuted by a processor associated with a computing device, cause theprocessor to perform the stages described. Additionally, the examplemethods summarized above can each be implemented in a system including,for example, a memory storage and a computing device having a processorthat executes instructions to carry out the stages described.

Both the foregoing general description and the following detaileddescription are exemplary and explanatory only and are not restrictiveof the examples, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart of an exemplary method for providing dynamicserviceability for an SDDC.

FIG. 2 is a system diagram of an SDDC that provides information tovarious destinations.

FIG. 3 is system diagram of an example serviceability module.

DESCRIPTION OF THE EXAMPLES

Reference will now be made in detail to the present examples, includingexamples illustrated in the accompanying drawings. Wherever possible,the same reference numbers will be used throughout the drawings to referto the same or like parts.

Examples described herein include systems and methods for providingdynamic serviceability for an SDDC. Although the examples describedherein refer to an SDDC, the systems and methods equally can be appliedto other types of data-centers. Serviceability of these data-centers,including SDDCs, can be provided by collecting information regarding thehardware and software components of the data-center, filtering andformatting the information to suit a particular partner entity's needs,and providing the filtered and formatted information to the partnerentity. Some example systems can utilize an inference engine toautomatically diagnose issues and implement fixes without requiringmanual intervention.

FIG. 1 provides a flowchart of an example method for providing dynamicserviceability for an SDDC. FIG. 2 provides a system diagram of an SDDCthat provides information to various destinations. FIG. 3 providessystem diagram an example serviceability module that can be implementedin an SDDC to provide dynamic serviceability.

Turning to the example method of FIG. 1, stage 110 can includecollecting data-center information from a management service thatmonitors the SDDC. The data-center information can include anyinformation regarding the hardware and software components that make upthe SDDC. With respect to a particular hardware component, for example,the data-center information can include an identification of the type ofhardware component, its manufacturer, build date, installation date,processor specification, memory specification, firmware, software,operating system (“OS”) type and version, device identification number,workload, number of supported virtual machines (“VMs”), configurations,logs, and any other relevant information about the hardware component.

Software components can also be described by the data-centerinformation. For example, with respect to a particular softwarecomponent, the data-center information can include the current softwareversion, previous software version, previous patches, available updates,hardware compatibility, compatibility with other software components orwith partner entity support sites, required resources to ensureperformance, number and types of supported or currently running VMs,configuration information, logs, and any other relevant informationabout the software component.

The data-center information can also include information regardingevents, alerts, and alarms for any component in the SDDC. An event canbe a record of a user action or system action that occurs on an objectin a server, host, or other component of the SDDC. Actions that might berecorded as events include but are not limited to: a license keyexpiring, a VM being powered on or off, a user logging into or out of aVM, and a host connection being lost or reestablished.

Data-center information regarding an event can include details about theevent, such as who or what generated it, when it occurred, and what typeof event it is. In one example, there are three types of events: aninformation event, a warning event, and an error event. An informationevent can simply provide information that does not require a response,such as, for example, a host connection being established orreestablish. A warning event can provide a warning but not necessarilyindicate that immediate action is required. For example, a warning eventcan include a warning that a host is operating at over 90% capacity andmay exceed capacity in the near future. An error event can provideinformation regarding an error that needs to be corrected, such anindication that a host has unexpected powered down or lost connection,or that a license has expired and caused a host to cease operations.

Alarms can be notifications activated in response to an event, a set ofconditions, or the state of an inventory object. An alarm can includevarious types of information that are all considered “data-centerinformation.” For example, an alarm can include a name and descriptionthat provides an identification of the source of the alarm. An alarm canalso include an alarm type that defines the type of object that isrelevant to the alarm. It can include triggers that define the event,condition, or state that triggers the alarm and defines the notificationseverity.

An alarm can further include tolerance thresholds that provideadditional restrictions on condition and state triggers thresholds thatmust be exceeded before the alarm is triggered. It can also includeactions that define operations to occur in response to a triggeredalarm. Actions can be predefined and specific to inventory object types,in some examples. Alarms can provide severity information according aseverity scheme that defines levels of severity. For example, an alarmcan display as green to indicate normal operation, yellow to indicate a“warning” state, and red to indicate an “alert” state. Alarms can beinitiated by a virtual infrastructure manager, which is described withrespect to FIG. 2.

In addition to events and alarms, the data-center information can alsoinclude alerts. An alert can be a record of a fault detected in thesystem. For example, an alert can be based on an alarm that describes asystem failure. An SDDC manager—described in more detail with respect toFIG. 3—can generate an alert based on a determination that a fault hasoccurred. In some examples, a single occurrence can cause both an alarmand an alert to be generated by the system. The alarm can be generatedby a virtual infrastructure manager, while the alert can be generated byan SDDC manager. In some examples, either or both of the virtualinfrastructure manager and SDDC manager can generate either or both ofthe alarm and alert, as desired.

At stage 120, the system can filter the data-center information based onpredetermined rules. The predetermined rules can be provided by apartner entity in some examples. Because the collection of data-centerinformation can be wide ranging, a large amount of data can be gatheredand needs to be sorted into usable information. For example, in somecircumstances duplicate events, alarms, or alerts can be generated basedon a single event. This can occur when multiple detection components,such as a virtual infrastructure manager and an SDDC manager, aremonitoring the same system component. The predetermined rules canindicate that the duplicate events, alarms, and alerts be removed fromthe data-center information. The rules can also recognize events,alarms, and alerts that are related to one another, and combine them toform a single telemetry element.

The filtering at stage 120 can also be performed based on apredetermined list of metrics provided by a partner entity. For example,a partner entity may not want to receive any alarms that are given agreen color code (indicating normal operation of a component). In thatexample, the partner entity's predetermined list can include yellow- orred-coded alarms. When a green-coded alarm is present in the data-centerinformation, the filtering process can remove the information related tothat alarm based on the predetermined list. Similar rules or indicationscan be provided, either by the partner entity or otherwise, dictatingthe type of filtering to be performed at this stage. For example,filtering can further include purging irrelevant information or groupingrelated information items together.

The various components in an SDDC can be serviced by multiple differentpartner entities, each of which can use their own processes and systemsfor handling issues that arise in the SDDC. Each partner entity canspecify a particular format for data-center information it receives. Forexample, a partner entity can specify that the data-center informationshould be formatted to include a field specifying the component atissue, followed by a field specifying the type of event, alarm, or alertrelevant to that component, followed by the specifications of thatcomponent, and so on. As explained with respect to FIGS. 2 and 3, apartner entity can provide formatting instructions through aserviceability configuration interface that provides a mechanism forautomatically receiving and implementing such instructions at the SDDC.These formats can be arbitrarily chosen by a partner entity and can varyfrom one partner entity to another.

Stage 130, therefore, can include translating the filtered data-centerinformation into a partner-specific format requested by the partnerentity. This stage can include identifying the relevant partner entityimplicated by the data-center information at issue, obtaininginstructions associated with that partner entity regarding the requestedformat of information, and formatting the information to match therequested format.

Even after filtering and translating, the translated data-centerinformation is not necessarily provided directly to the relevant partnerentity. Instead, the translated data-center information can be packagedand transmitted to the partner entity as a batch. Stage 140 can includegenerating any necessary metadata associated with the translateddata-center information to prepare it for transmission to the partnerentity. This can include, for example, generating metadata indicatingthe dates and times covered by the data-center information, thefiltering or translating rules applied to the data-center information,or any other relevant information describing the translated data-centerinformation.

The metadata and translated data-center information can be transmittedto the relevant partner entity at stage 150. This stage can includeidentifying a destination associated with a partner entity and ensuringthat the partner entity matches a partner entity previously associatedwith the data-center information. The stage can further includeattempting to connect a partner site associated with the partner entity,and after connecting, attempting to upload the batched metadata andtranslated data-center information. Stage 150 need not includesuccessfully transmitting the batched data to the partner entity. In theevent of an unsuccessful transmission, the method can continue to stage160.

Stage 160 can be performed in response to determining that the partnersite is not available. In that case, stage 160 can include forwardingthe metadata and translated data-center information to apartner-configured storage location. The partner-configured storagelocation can be a cloud storage location configured by, and accessibleto, the partner entity. The partner entity can then retrieve themetadata and translated data-center information when it can do so.

To alert the partner site that the transmission was unsuccessful andthat the metadata and translated data-center information is stored at apartner-configured storage location, stage 170 can be carried out. Stage170 can include generating an alert regarding downtime of the partnersite. For example, the alert can indicate the time of any attempts totransmit the data to the partner entity site. The alert can alsoindicate the size of the attempted transmission and other identifyinginformation. Additionally, stage 170 can include identifying thedestination of the metadata and translated data-center information. Thiscan include, for example, identifying a cloud storage location at whichthe information is stored. The partner entity can utilize thisinformation to retrieve the information and resolve any issues with thepartner site that may be needed.

FIG. 2 provides an illustration of an example system capable of carryingout the method of FIG. 1 and other methods described herein. FIG. 2depicts an SDDC 200 that can provide data-center information to variousdestinations through a network 238. The destinations can include, forexample, a first partner entity 240, second partner entity 250, cloudanalytics service 260, multi-site SDDC management location 270, and acloud storage service 280. These are discussed in more detail below.

The SDDC 200 can include any number of physical servers 202 that providefunctionality for storing, retrieving, and transmitting information.Each physical server 202 can include, for example, one or moreprocessors, one or more storage drives, and memory. The servers 202 canbe traditional servers in one example. In another example, one or moreservers 202 are HCl-capable and include direct-attached storage. Theservers 202 can be off-the-shelf components capable of providing aplug-and-play solution to creating a pool of server resources. Forexample, a server 202 can utilize its own direct-attached storage aswell as the direct-attached storage of other servers 202. The storage,backup, and retrieval processes can be software-controlled processes.The servers 202 can each run multiple VMs that are also softwarecontrolled. Although referred to as “servers,” the servers 202 caninclude any SDDC hardware, including servers, switches, gateways,routers, bridges, modems, hubs, repeaters, or other networking hardware.

The servers 202 can communicate with any number of hardware plugins 204that interact with the physical hardware of the servers 202. Thehardware plugins 204 can provide a bridge to a hardware managementservice (“HMS”) 208 and allow the HMS 208 to manage the servers 202. Forexample, the HMS 208 can provide functions required for discovering,bootstrapping, and monitoring the servers 202 and other hardware. Insome examples, the hardware plugins 204 can be provided by a vendor ormanufacturer of a server 202 component. In some examples, the hardwareplugins 204 can be provided by the entity providing or managing theSDDC. The plugins 204 can allow the HMS 208 to access all servers andswitches 202 in the system and send and receive information to and fromthe servers and switches 202.

The HMS 208 can provide information obtained from the plugins 204 to amonitoring module 214. The monitoring module 214 can include a varietyof agents that implement different mechanisms to collect data. Theagents can be generated based on code executing on a computing device.In some examples, an ESXi agent can use a Common Information Model(“CIM”) to collect data, a fabric agent can use Storage ManagementInterface Specifications (“SMI-S”) to collect data, and a storage arrayagent can use either SMI-S or an Application Programming Interface(“API”) such as VMWARE's API for Storage Awareness (“VASA”). Themonitoring module can communicate with both a serviceability module 222and a virtual infrastructure manager(s) 212.

The virtual infrastructure manager 212 can be a virtualization platform,such as VMWARE's vSPHERE, that allows a manager to run, manage, connect,and secure virtual resources in the system. In some examples, thevirtual infrastructure manager 212 includes multiple virtualizationplatforms, such as multiple instances of vSPHERE. The virtualinfrastructure manager 212 can collect events, alarms, and alerts at thecluster level. In some examples, the virtual infrastructure manager 212can collect hardware-level alerts, but typically those hardware-levelalerts are not visible to the virtual infrastructure manager 212. Thevirtual infrastructure manager 212 can be one source of events, alarms,and alerts that are processed by the serviceability module 222.

The virtual infrastructure manager 212 can also communicate with legacysoftware services 206 and management and workload VMs 210. The legacysoftware services 206 can be used when, for example, a partner entitywishes to rely on its own software solutions for collectingserviceability information. For example, a partner can rely on theircustom firmware that collects metrics, events, alarms, and alerts fromthe relevant hardware servers 202. The legacy software services 206 canthen transmit the collected information to the virtual infrastructuremanager 212.

The management and workload VMs 210 can include one or more VMsdedicated to managing the workloads of various VMs and associatedoperating in the SDDC. For example, the management and workload VMs 210can track the number of VMs on a particular host, the memory andcomputing utilization of the host, license utilization for the host, andany other metrics associated with the VMs operating in the SDDC. Themanagement and workload VMs 210 can provide information to the virtualinfrastructure manager 212 and can carry out instructions received fromthe virtual infrastructure manager 212. For example, the virtualinfrastructure manager 212 can instruct the management and workload VMs210 to rebalance a set of VMs.

The virtual infrastructure manager 212 can provide any collectedinformation to both a virtual rack manager 218 and SDDC services 216. Insome examples, the virtual infrastructure manager 212 can storeinformation at a storage location 220 accessible to both the virtualrack manager 218 and SDDC services 216. The virtual rack manager 218 canrun on a cluster of server 202 nodes, in one example. In one example,the virtual rack manager 218 can use a management kernel to communicatewith the servers 202 through one or more spine switches. The virtualrack manager 218 can instruct the virtual infrastructure manager 212 toperform management operations on hardware associated with the servers202. For example, the virtual rack manager 218 can instruct the virtualinfrastructure manager 212 to restart a server or switch.

The SDDC services 216 can be part of, or utilized by, an SDDC manager.The SDDC manager can be a hybrid cloud platform such as VMWARE's CLOUDFOUNDATION, for example. The SDDC manager can provide integrated cloudinfrastructure, including compute, storage, networking, and security,and cloud management services to run applications in private and publicenvironments. The SDDC services 216 can be used to collect informationfrom the hardware servers and switches (Top of the Rack(ToR) Switches,Spine Switches, Management Switches) 202 that would otherwise not becollected by the virtual infrastructure manager 212. For example, asmentioned above, the virtual infrastructure manager 212 might not beable to view or detect hardware-level events, alerts, or alarmsgenerated by the servers and switches 202. In that example, one or moreSDDC service 216 can detect these hardware-level events and provide themto the serviceability module 222.

The serviceability module 222 can collect all metrics, events, alerts,and alarms from both the virtual infrastructure manager 212 and the SDDCservices 216 and perform various operations based on the collectedinformation. The serviceability module 222 can include a serviceabilityplugin interface 224 that provides a partner entity 240, 250 with one ormore interfaces to customize a wide variety of features and options. Forexample, the serviceability plugin interface 224 can include aconfiguration interface 226, data interface 228, and operationsinterface 230, all of which can be made accessible to one or morepartner entities 240, 250.

The configuration interface 226 can allow a partner entity 240, 250 toconfigure the serviceability module 222 in various ways. For example, apartner entity 240, 250 can utilize the configuration interface 226 toconfigure the partner-specific specification or format in which thepartner wants to receive serviceability information. This includes, forexample, the particular statistics, events, alerts, and alarms a partnerentity wishes to receive from the serviceability module 222. As anotherexample, the configuration interface 226 can be used to configure apartner-specific filtering to be done on the serviceability information.This can include, for example, filtering out non-critical alarms oralarms relating to a particular hardware unit. In another example, theconfiguration interface 226 can be used to configure a collection pollcycle time period for collecting serviceability information fromdifferent sources. The configuration interface 226 can also provide forconfiguration of collection rules and parameters for rules andmachine-learning models to be implemented at an inference engine 234(discussed in more detail later).

The data interface 228 can be an interface responsible for forwardedprocessed serviceability information to the relevant partner channel.For example, the data interface 228 can add metadata to theserviceability information (as discussed with respect to stage 140 ofFIG. 1) and send the information in a batch to the partner entity 240,250. If the data interface 228 is unable to reach a partner entity 240,250, it can forward the metadata and serviceability information to acloud storage service 280. It can also alert the partner entity 240, 250regarding the failed transmission and provide identifying informationfor the cloud storage service 280 where the information is stored andretrievable by the partner entity 240, 250.

The operations interface 230 is an interface that can carry outexecutive functionalities on the SDDC 200 based on input from a partnerentity 240, 250 or from the inference engine 234. In one example, analert is received by the partner entity 240, 250 through the datainterface 228, indicating that a firmware patch is incompatible with thevirtual infrastructure manager 212 for a particular workload domain. Thepartner entity 240, 250 can determine that a new patch version wouldsolve the issue. The partner entity 240, 250 can then utilize theoperations interface 230 to trigger the necessary patch upgrade, whichcan cause the required commands to be provided to the SDDC manager toimplement. As another example, the operations interface 230 can beutilized to recover from issues or fault conditions. For example, if analert indicates that the hardware management service 208 is not running,the operations interface 230 can be used to transmit a command to thevirtual rack manager 218 to start or restart the hardware managementservice 208.

The serviceability module 222 can also include a serviceabilitytelemetry processor 232 that includes various components for collecting,filtering, and translating serviceability information originating fromthe virtual infrastructure manager 212 or SDDC manager. These componentsof the serviceability telemetry processor 232 are depicted and discussedin more detail with respect to FIG. 3.

The serviceability module 222 can also include an inference engine 234,as mentioned above. The inference engine 234 can be a component thatapplies logical rules to information in order to deduce new information.The inference engine 234 can apply these logical rules to anyserviceability information gathered from the SDDC 200. The logical rulesapplied by the inference engine 234 can be provided by a partner entity240, 250, causing the inference engine 234 to apply those custom rulesto information that is relevant to the particular partner entity 240,250 that provided the rules.

A partner entity 240, 250 can create custom rules for the inferenceengine 234 for various types of actions. Custom rules can affect thegeneration of statistics, such as by defining the triggering events thatcause a particular statistic to be generated. For example, a custom rulecan specify that an indication of normal operation need not generate astatistic for further processing. In other examples, custom rules can bedirected to rules for the collection of information (such as events,alarms, and alerts), filtering of information, clustering ofinformation, correlations of information, classification of information.Custom rules can also be implemented for indicating a particular cloudstorage service 280 or location where information should be sent whenthe partner site is unavailable. The rules can further specify thegeneration of an alert regarding the downtime of the partner site andthe respective destination of the information that was not successfullytransmitted.

In another example, the inference engine 234 can utilizepartner-specified rules to predict the particular partner site to whicha data transmission should be directed. For example, the inferenceengine 234 can create a telemetry processor workflow based on theparticular partner plugins 246, 256 registered at the serviceabilitymodule 222. In one example, a partner support service 244, 254determines that it wants to receive notifications regarding a set ofvirtual infrastructure manager alarms. The partner support service 244,254 can provide communication endpoints such as Uniform ResourceLocators (“URLs”), Internet Protocol (“IP”) address, ports, and so on.

While the inference engine 234 can utilize static rules, it can alsoperform machine-learning processes as specified by one or more rules.For example, a partner entity 240, 250 can push a machine-learning modelat the configuration interface 226 for use by the inference engine 234.In one example, the machine-learning model can be applied to determinenew issues that have not yet produced an expected event, alert, oralarm. In another example, the machine-learning model can be applied topredict the partner site to which a data transmission should be directedwhen there is not a static rule to make such a determination. Forexample, a first static rule can indicate that alerts from componentsA1, A2, and A3 should be provided to a first partner entity 240, whichalerts from components B1, B2, and B3 should be provided to a secondpartner entity 250. When an alert is received from component A4, amachine-learning model can be applied to predict that the alert shouldbe provided to the first partner entity 240 based on the patterns setforth in the static rules. This example is simplified for the sake ofexplanation. The machine-learning models need not be limited toexamining other static rules, but instead can examine any informationmade available to it by the serviceability module 222.

The serviceability module 222 can also include a scheduler 236 that isresponsible for triggering a serviceability telemetry processorworkflow. For example, when an event, alert, or alarm is received at theserviceability module 222, the scheduler 236 can spawn a serviceabilitytelemetry processor workflow that includes a particular poll cycle, acollection step, a filtering step, and a translation step. The scheduler236 and serviceability telemetry processor workflow are described inmore detail with respect to FIG. 3.

The serviceability module 222 can transmit information through a network238, such as the Internet, to various destinations. In some examples,the information can be transmitted to a partner entity 240, 250. Thepartner entity 240, 250 might be responsible for troubleshooting issuesassociated with hardware components of the SDDC 200, for example. Apartner entity 240, 250 can utilize a partner plugin 246, 256 thatallows the data interface 228 of the serviceability module 222 toconnect to the partner site and transmit or receive information to orfrom the partner site. A partner entity 240, 250 can also include apartner support service 244, 254 that examines incoming information andprovides instructions, solutions, or other support to the SDDC 200. Thepartner support services 244, 254 can include one or more partnersupport people 242, 252.

The serviceability module 222 can also transmit information to a cloudanalytics service 260. The cloud analytics service 260 can be utilizedto analyze information before it is sent to a partner entity 240, 250.The cloud analytics service 260 can be used to build a knowledge baseregarding any components of the SDDC 200 or implementations of a partnerentity 240. An example cloud analytics service 260 is the VMWAREANALYTICS CLOUD, although other cloud analytics services can be used aswell. The cloud analytics service 260 can include a plugin 264 fortransmitting information as well as the analytics service 262.

Additionally, the serviceability module 222 can transmit information toa multi-site SDDC management location 270. The multi-site SDDCmanagement location 270 can include a plugin 274 for interfacing withthe serviceability module 222 as well as a management component 272. Theserviceability module 222 can transmit information to the multi-siteSDDC management location 270 regardless of whether the information istransmitted to a partner entity 240, 250, cloud storage service 280, orcloud analytics service 260. The multi-site SDDC management location 270can be used to gather information from multiple SDDCs 200 and managethem at one location, even where the SDDCs 200 are implemented indisparate locations.

FIG. 3 provides an example system diagram of the serviceability module222 discussed in FIG. 2. As shown in the drawing and explained withrespect to FIG. 2, the serviceability module 222 can communicate with apartner site through any of three interfaces: the data interface 228,the configuration interface 226, and the operations interface 230. Apartner entity 240, 250 can utilize the configuration interface 226 tocustomize the serviceability module 222 to suit the partner's particularneeds. For example, a partner entity can submit, through theconfiguration interface 226, configuration instructions regarding apartner-specific specification or format in which they want to receiveserviceability information. The configuration instructions can apply toany of the collecting, filtering, and translating stages performed bythe serviceability module 222. The configuration instructions can alsoapply to the inference engine 234, such as by providing static rules ormachine-learning models to be utilized by the inference engine 234.

The configuration instructions can be gathered by a configurationcontroller 302. The configuration controller 302 can be a component ofthe configuration interface 226 in one example. In another example, theconfiguration controller 302 is a portion of the serviceability module222 but operates as a process separate from the configuration interface226. The configuration controller 302 can collect any instructionsreceived through the configuration interface 226 and provide them to aconfiguration service 304.

The configuration service 304 can interpret the configurationinstructions collected by the configuration controller 302 and formatthe configuration instructions in a manner suitable for providing toother components of the serviceability module 222. For example, theconfiguration service 304 can receive an instruction to create a staticrule specifying that events, alerts, and alarms from a particularhardware device should be ignored for the next 30 minutes. Theconfiguration service 304 can generate an instruction for the inferenceengine 234, using a format understood by the inference engine 234, andthen forward the formatted instructions to the inference engine 234 tobe implemented.

The configuration service 304 can also save configuration information toa database 310 that is accessible to components within theserviceability module 222. The configuration information in the database310 can be used by other components of the serviceability module 222,such as a filter 318, 328 or collector 316, 326, as needed. Thecomponents of the serviceability module 222 can access the database 310whenever configuration information is required, such as when thescheduler 236 launches a workflow that implicates a configuration savedin the database 310.

Like the configuration interface 226, the operations interface 230 caninclude an operations controller 306 and an operations service 308. Theoperations controller 206 can receive operations information provided bya partner site through the operations interface 230. For example, apartner entity 240, 250 can provide a patch upgrade intended to solve anissue with a previous patch. The patch can be uploaded through theoperations interface 230 with instructions to install the patch onfirmware associated with a hardware unit. The operations controller 306can gather the patch and installation instructions and provide it to theoperations service 308.

In addition to receiving operations information through the operationsinterface 230, the operations controller 306 can also receive operationsinformation from the inference engine 234. For example, the inferenceengine 234 can use a machine-learning model to predict that a patchshould be applied to the firmware of a particular hardware unit.Expanding on that example, a previous issue with a first hardware unitmay have been solved by a partner entity 240, 250 providing a patchupdate specific to the first hardware unit. When the same issue ariseswith a second hardware unit that is associated with the same partnerentity 240, 250, the inference engine 234 can use a machine-learningmodel provided by that partner entity 240, 250 to determine that asimilar patch should be applied to the second hardware unit. Certainaspects of the patch can be changed to suit the second hardware unit,such as by updating identification information calling out the secondhardware unit. The inference engine 234 can generate the instructions,and in some examples the patch itself, and provide one or both items tothe operations controller 306. The operations controller 306 can receivethese items and provide them to the operations service 308 forexecution.

Regardless of the upstream source of the instructions, the operationsservice 308 can receive instructions from the operations controller 306,format them, and deliver them to the SDDC manager 314. The operationsservice 308 can utilize an API call, such as a representational statetransfer (“REST”) API call, that causes the SDDC manager 314 to retrievethe formatted instructions and carry them out accordingly. For example,the SDDC manager 314 can instruct the virtual rack manager 218 orhardware management service 208 to install a custom patch to thefirmware of a hardware unit.

The inference engine 234 can also apply rules that dictate the actionsof a scheduler 236. The scheduler 236 can responsible for triggering aserviceability telemetry processor workflow. For example, when an event,alert, or alarm is received at the serviceability module 222, thescheduler 236 can spawn a serviceability telemetry processor workflowthat includes a particular poll cycle, a collection step, a filteringstep, and a translation step. The inference engine 234 can provide rulesfor determining when to spawn a serviceability telemetry processorworkflow, determining a poll time to apply to the collection step of theworkflow, determine whether to involve the SDDC manager 314 or virtualinfrastructure manager 212 in the workflow, and any other variablesrelated to generating a workflow.

In the example of FIG. 3, the scheduler 236 has spawned twoserviceability telemetry processor workflows—one based on informationfrom the SDDC manager 314 and one based on information from the virtualinfrastructure manager 212. The scheduler 236 can determine the type ofworkflow based on the event, alert, alarm, or rule triggering theworkflow. For example, if an alert is received from the SDDC services216 indicating a hardware failure, the scheduler 236 can apply a rule(provided by the inference engine 234 in some examples) to spawn aworkflow that includes the SDDC manager 314. Similarly, if an alarm isreceived from the virtual infrastructure manager 212, the scheduler 236can apply a rule to spawn a workflow that includes the virtualinfrastructure manager 212. The scheduler can set poll times based onthe specific workflow. In one example, the hardware management service208 uses a poll time of 10 minutes, so a workflow involving the hardwaremanagement service 208 can include the SDDC manager 314 and use a polltime of 10 minutes.

Each workflow spawned by the scheduler 236 can include implementationsof a collector 316, 326, filter 318, 328, and translator 320, 330. Theparticular implementations can be based on the source of the triggeringevent, alarm, or alert. For example, an alert originating from the SDDCservices 216 can spawn a workflow that involves the SDDC manager 314 andutilizes a collector 316 tailored to collecting information from theSDDC services 216 and hardware management service 208, such as by usinga REST API interface to poll alerts. Similarly, an alert originatingfrom the virtual infrastructure manager 212 can spawn a workflow thatutilizes a collector 326 tailored to collecting information from thevirtual infrastructure manager 212. For example, the collector 326 canuse a virtual infrastructure API such as the vSPHERE API to poll alarms.

The collectors 316, 326 can route the collected events, alarms, andalerts to the respective output pipelines. The output pipelines can bedifferent for each distinct combination of source and endpoint. Theoutput pipelines can include, for example, filters 318, 328 andtranslators 320, 330. A partner entity 240, 250 can specify a particularcollector to use for particular events, alarms, and alerts, by providingthe required information through the configuration interface 226. Inthat example, when a workflow is spawned by the scheduler, the collector316, 326 can be selected based on configuration information stored inthe configuration database 310.

The collector 316, 326 can provide the collected events, alarms, andalerts to an appropriate filter 318, 328. The filter 318, 328 canperform a filtering process on the collected information, removing anyinformation that is not suitable for the partner entity 240, 250relevant to the workflow pipeline. The filter 318, 328 can perform thefiltering by, for example, applying configuration rules stores in thedatabase 310 associated with the relevant partner entity 240, 250. Theconfiguration rules can be stored in the database 310 as a result ofreceiving the configuration rules through the configuration interface226, as described earlier. The filter 318, 328 can also perform thefiltering based on rules inherent to the filter 318, 328, such asremoving duplicate events, alerts, and alarms. The filter 318, 328 canfurther remove any information that applies to other partner entities240, 250 rather than one of interest for the particular workflowprocess.

After performing the filtering operation, the filter 318, 328 canprovide the filtered information to the relevant translator 320, 330 inthe workflow process. As explained with respect to FIG. 1, each partnerentity 240, 250 can specify a particular format for data-centerinformation it receives. For example, a partner entity 240, 250 canspecify that the data-center information should be formatted to includea field specifying the component at issue, followed by a fieldspecifying the type of event, alarm, or alert relevant to thatcomponent, followed by the specifications of that component, and so on.A partner entity 240, 250 can provide formatting instructions throughthe configuration interface 226, which can be processed by theconfiguration controller 302 and configuration service 304 before beingstored in the configuration database 310. The translator 320, 330 cantherefore access the configuration database 310 and retrieve therelevant translation instructions required for a particular partnerentity 240, 250.

After carrying out the relevant translation instructions, the translator320, 330 can provide the translated data-center information to the datainterface 228. The data interface 228 can package the information into abatch and generate any necessary metadata associated with the translateddata-center information to prepare it for transmission to the partnerentity 240, 250. This can include, for example, generating metadataindicating the dates and times covered by the data-center information,the filtering or translating rules applied to the data-centerinformation, or any other relevant information describing the translateddata-center information. The data interface 228 can then transmit theinformation a partner entity 240, 250, cloud storage service 280, cloudanalytics service 260, multi-site SDDC management location 270, or somecombination thereof.

The data interface 228 can provide collection metadata to theconfiguration database 310, allowing the database 310 to keep a completerecord of serviceability. The collection metadata can provideinformation sufficient to determine the continuity or sequence of thetelemetry data processed by the serviceability module 222. Thecollection metadata can include, for example, a collection timestampthat indicates the time that a particular set of data was collected andsuccessfully published to the partner site. This can be used to identifyany unpublished data from a previous poll cycle, for example.

The serviceability module 222 can implement solutions or performmaintenance procedures with respect to one or more components of theoverall SDDC 200 system. The solutions or maintenance can include, forexample, a software or firmware update of a component, replacement of acustomer-replaceable unit such as a hard disk, solid-state drive, orpower unit, replacement of a field-replaceable unit such as a server,CPU, or memory, a workload domain delete operation, or a cluster deleteoperation.

When these solutions or maintenance procedures are occurring, theparticular component can produce additional events, alerts, or alarms byvirtue of being shut down, restarted, changed, or otherwise affected bythe procedure. These events, alerts, and alarms can be consideredredundant or irrelevant, as they are produced in connection with aplanned improvement, fix, or upgrade. The serviceability module 222 can“mute” the affected component while work (solution or maintenance) onthe component is ongoing.

In one example, the serviceability module 222 mutes the relevantcomponent by instructing the virtual infrastructure manager 212 orhardware management service 208 to ignore events, alerts, and alarmsgenerated by the relevant component for a predetermined period of time,or until instructed otherwise. In another example, the component ismuted by a rule disabling collection of events, alerts, and alarms bythe collector 316, 326 during the relevant time period. Similarly, inanother example, the component can be muted by a rule requiring thefilter 318, 328 to filter out any events, alerts, and alarms originatingfrom the component during the relevant time period. The muting processcan apply to any component, such as a deployment, domain, cluster, host,server, or other component.

Other examples of the disclosure will be apparent to those skilled inthe art from consideration of the specification and practice of theexamples disclosed herein. Though some of the described methods havebeen presented as a series of steps, it should be appreciated that oneor more steps can occur simultaneously, in an overlapping fashion, or ina different order. The order of steps presented are only illustrative ofthe possibilities and those steps can be executed or performed in anysuitable fashion. Moreover, the various features of the examplesdescribed here are not mutually exclusive. Rather, any feature of anyexample described here can be incorporated into any other suitableexample. It is intended that the specification and examples beconsidered as exemplary only, with a true scope and spirit of thedisclosure being indicated by the following claims.

What is claimed is:
 1. A method for providing dynamic serviceability fora software-defined data center (SDDC) to a partner entity, comprising:collecting data-center information from a management service thatmonitors at least a portion of the SDDC; filtering the data-centerinformation based on a stored list of metrics; translating the filtereddata-center information into a partner-specific format requested by thepartner entity; generating metadata associated with the translateddata-center information; and transmitting the metadata and translateddata-center information to a partner site associated with the partnerentity.
 2. The method of claim 1, further comprising: in response todetermining that the partner site is not available, forwarding themetadata and translated data-center information to a partner-accessiblestorage location; and in response to determining that the partner sitehas recovered, generating an alert regarding downtime of the partnersite and identifying the destination of the metadata and translateddata-center information.
 3. The method of claim 1, wherein the storedlist of metrics is provided by the partner entity through aconfiguration interface and stored in a configuration database.
 4. Themethod of claim 1, wherein the stages are performed based on at leastone rule supplied by the partner entity through a configurationinterface and enforced by an inference engine.
 5. The method of claim 1,wherein collecting data-center metrics is performed according to a polltime, and wherein the poll time is dynamically set by an inferenceengine based on settings provided by the partner entity.
 6. The methodof claim 1, further comprising: receiving a response from the partnerentity through a serviceability operations interface; and implementing asolution at the SDDC, based on the received response, without manualintervention, wherein the solution is implemented automatically based onthe inference engine detecting the problem at the SDDC and executing oneor more rules provided by the partner entity through the serviceabilityoperations interface.
 7. The method of claim 6, wherein implementing thesolution further comprises: identifying a component that requires thesolution; and disabling the collecting of metrics from the componentuntil the solution is implemented.
 8. A non-transitory,computer-readable medium comprising instructions that, when executed bya processor of a computing device, cause the processor to perform stagesfor providing dynamic serviceability for a software-defined data center(SDDC) to a partner entity, the stages comprising: collectingdata-center information from a management service that monitors at leasta portion of the SDDC; filtering the data-center information based on astored list of metrics; translating the filtered data-center informationinto a partner-specific format requested by the partner entity;generating metadata associated with the translated data-centerinformation; and transmitting the metadata and translated data-centerinformation to a partner site associated with the partner entity.
 9. Thenon-transitory, computer-readable medium of claim 8, the stages furthercomprising: in response to determining that the partner site is notavailable, forwarding the metadata and translated data-centerinformation to a partner-accessible storage location; and in response todetermining that the partner site has recovered, generating an alertregarding downtime of the partner site and identifying the destinationof the metadata and translated data-center information.
 10. Thenon-transitory, computer-readable medium of claim 8, wherein the storedlist of metrics is provided by the partner entity through aconfiguration interface and stored in a configuration database.
 11. Thenon-transitory, computer-readable medium of claim 8, wherein the stagesare performed based on at least one rule supplied by the partner entitythrough a configuration interface and enforced by an inference engine.12. The non-transitory, computer-readable medium of claim 8, whereincollecting data-center metrics is performed according to a poll time,and wherein the poll time is dynamically set by an inference enginebased on settings provided by the partner entity.
 13. Thenon-transitory, computer-readable medium of claim 8, the stages furthercomprising: receiving a response from the partner entity through aserviceability operations interface; and implementing a solution at theSDDC, based on the received response, without manual intervention,wherein the solution is implemented automatically based on the inferenceengine detecting the problem at the SDDC and executing one or more rulesprovided by the partner entity through the serviceability operationsinterface.
 14. The non-transitory, computer-readable medium of claim 13,wherein implementing the solution further comprises: identifying acomponent that requires the solution; and disabling the collecting ofmetrics from the component while the solution is being implemented. 15.A system for providing dynamic serviceability for a software-defineddata center (SDDC) to a partner entity, comprising: a memory storageincluding a non-transitory, computer-readable medium comprisinginstructions; and a computing device including a processor that executesthe instructions to carry out stages comprising: collecting data-centerinformation from a management service that monitors at least a portionof the SDDC; filtering the data-center information based on a storedlist of metrics; translating the filtered data-center information into apartner-specific format requested by the partner entity; generatingmetadata associated with the translated data-center information; andtransmitting the metadata and translated data-center information to apartner site associated with the partner entity.
 16. The system of claim15, the stages further comprising: in response to determining that thepartner site is not available, forwarding the metadata and translateddata-center information to a partner-accessible storage location; and inresponse to determining that the partner site has recovered, generatingan alert regarding downtime of the partner site and identifying thedestination of the metadata and translated data-center information. 17.The system of claim 15, wherein the stored list of metrics is providedby the partner entity through a configuration interface and stored in aconfiguration database.
 18. The system of claim 15, wherein the stagesare performed based on at least one rule supplied by the partner entitythrough a configuration interface and enforced by an inference engine.19. The system of claim 15, wherein collecting data-center metrics isperformed according to a poll time, and wherein the poll time isdynamically set by an inference engine based on settings provided by thepartner entity.
 20. The system of claim 15, the stages furthercomprising: receiving a response from the partner entity through aserviceability operations interface; and implementing a solution at theSDDC, based on the received response, without manual intervention,wherein the solution is implemented automatically based on the inferenceengine detecting the problem at the SDDC and executing one or more rulesprovided by the partner entity through the serviceability operationsinterface.